Training transfer : a bridge between the theory - oriented and product - oriented approaches to ITS design

The design and implementation of intelligent tutoring systems is mainly oriented towards scientific goals. Implementation is viewed as a methodology for research and systems are considered as its byproduct. However, this contribution shows that the theoretical grounds of our discipline do not restrict its applicability to theoretical advances. They respond instead to the trainer's concern about the real effectiveness of training, as measured by transfer. We sketch out the computational and psychological basis for developing systems which enhance transfer. We describe two approaches. The first reduces the difficulty of transfer by maximizing source-target behavioural overlapping. The second aims to develop transfer skills by increasing multiple and reflective access to knowledge. We relate these approaches to the functionalities expected from an intelligent training system. Dillenbourg P., Hilario M., Mendelsohn P. & Schneider D. (1990) Training transfer: A bridge between the theory-oriented and product-oriented approaches of ITS design. Actes du Second Congrès Européen "Multi-Media, Intelligence Artificielle et Formation" (APPLICA-90). Lille (France), septembre 1990 Dillenbourg et al. Training Transfer Draft for APPLICA 90 p.2 Introduction The design of intelligent tutoring systems (ITSs) is generally presented as a research domain rather than as a engineering problem. In this domain, the implementation process (i.e. the engineering aspects) is viewed as a methodology. Wenger expressed very well this position : "Teaching is difficult, and the fact that designing intelligent tutoring systems requires such a deep understanding of the processes involved may well be a sign that we have found a methodology whereby we can start attacking general issues in a systematic way" (Wenger, 1987, p.8). This statement represents the very scientific view of ITS research : implementing systems contributes to an increase of the knowledge corpus in the various disciplines affiliated to cognitive science, mainly artificial intelligence and cognitive psychology. Systems exist indeed as a by-product of research. We name this view of research the theory-oriented approach. Within this approach, most teams adopt a depth-first strategy : research efforts are focussed on a few specific components of the system at the expense of others. Subsequently, many systems never reach a level of completeness required for use with real learners, in the classrooms or for in-service training. These two first paragraphs should convince any staff manager that developing ITSs is a waste of time and money and that it is safer to carry on with traditional computer-based training (CBT) software. This would be a serious mistake. Between the ultra-sophisticated systems developed in some laboratories and the archaic courseware used in some companies, there is place for a product-oriented approach to ITS. Beside the latest advances, AI offers more secure techniques which can be applied to ITS design. The resulting systems do not constitute a major breakthrough in cognitive science but represent a real trade-off with respect to CBT. This approach inverses the means-ends relationship defined in the scientific approach : implementation shifts from a research methodology to a goal for development. The purpose of this paper is to describe in which way cognitive science may improve the quality of training. An exhaustive description of the potential benefits is beyond the scope of this paper. We do not tackle for instance the issue of reducing the costs of software development and maintenance by importing the experience acquired in the field of expert systems and knowledge acquisition. We concentrate on the skills acquired by the user through the use of an ITS, especially on the trainee's ability to transfer these skills to her professional tasks. The exploration of the relationship between cognitive science and its products aims to transcend the manicheism that characterizes this introduction. Our team does rather theory-oriented research, but aims to show how theoretical grounds support effective training actions. Dillenbourg et al. Training Transfer Draft for APPLICA 90 p.3 2. Effectiveness and transfer. Quality of training should not be measured by pre-test / post-test comparisons. Let's say brutally that a training system (computerized or not) is effective if trainees become better in doing their job ! There may exist a large distance between the behaviour measured at the end of training (source behaviour) and the professional behaviour (target behaviour). Research on human cognition showed that this transfer does not occur spontaneously, simply because source and target behaviours require the activation of partly different bodies of knowledge (Mayer,1988). The overlapping of the source and target bodies is inversely proportional to the distance of transfer and hence to its difficulty. The corollary of this law on transfer is that the real effectiveness of training systems is increased if we increase source-target behavioural overlapping. This is not a novelty : jumbo pilots are not trained on helicopters. They use instead flight simulators which reproduce as closely as possible the characteristics of the target situation. AI techniques have proved they offer the facilities required for simulating complex conditions. The stage of maximal overlapping is generally the last stage of learning and corresponds to maximal behavioural complexity. A simulation becomes a training system if it enables progressively tuning of overlap and complexity, if it provides the student with more information about her behaviour than the real world and if it constrains the trainee to reflect on her performance. This approach corresponds to AI work on analogy, especially with case-based reasoning (Riesbeck and Kolodner,1989) : the source case considered as the most similar to the target case is retrieved in order to minimize the task of adapting the source solution to the target problem. This principle of maximal overlapping includes its own limit. The target knowledge defines a class of behaviour and situations. This works well if the class is welldefined around a prototypical situation, but, in many cases, the trainer's life is not so easy. Often, the situation classes are open : the trainee is expected to perform fairly well in situations she never met during her training stage. In these cases, trainers cannot avoid the transfer problem by minimizing the source-target distance. Then, the issue is how ITSs can prepare the trainees to perform this transfer. Cognitive psychology brings partial answers to the ITS designer. 3. Knowledge and behaviour. There is some convergence among researchers in AI and in psychology about the fact that two types of knowledge may be identified. Let's take a classical example in diagnosis. The problem is to diagnose the cause X of an engine's breakdown, with symptoms {a,b,c}. The subject has some representation of the engine. This model describes the structure of the engine, its component, their function, their connections,etc. If the subject never met the symptoms {a,b,c}, she will use this model to perform complex causal reasoning until X is found. Inversely, if {a,b,c} is repeatedly associated to X, the subject will progressively induce a rule (X => {a,b,c}). Then, on the next occurrence of {a,b,c}, the diagnostic will be produced without lengthy causal reasoning. The model is only weakly related to the produced diagnostic, while the diagnostic rules directly fixes the solution. Most of recent work in expert systems and machine learning is articulated around this distinction. It finds its origins in the work of Clancey (1983) for decompiling the metaknowledge encompassed in Mycin rules. It has been recognised through the terminology used in second generation expert systems with discriminations such as "deep" versus "shallow" knowledge (Steels,1989), "structural" versus "experiential" knowledge (Vanwel-kenhuysen, 1988), "constructive" versus "classificative" Dillenbourg et al. Training Transfer Draft for APPLICA 90 p.4 problem solving (Van de Velde, 1988). The nuances between these distinctions are beyond the scope of this paper. We have to precise that the "deep" level also includes the knowledge needed to reason about the model. We will use the terms "conceptual" versus "behavioural" knowledge, simply to emphasize that the later determines behaviour (Dillenbourg and Self, to appear). Interestingly, an isomorphic distinction appeared in psychology. For instance, Richard's work on user's guides proposes a similiar distinction between the logic of functioning (how the device functions) and the logic of utilization (how to use the device)(Richard, 1984). The important point is to identify the respective advantages and short-comings of these knowledge levels, with respect to the transfer issue. The class of situations and behaviours covered by conceptual knowledge is obviously much larger than that covered by behavioural knowledge : the engine model is valid for any related diagnosis, even for other tasks such as dismantling the engine. The price of this extended scope is that conceptual knowledge is not prescriptive, i.e. it does not tell us what to do. These knowledge levels are complementary (they are probably not as distinct in human cognition than in computational forms). Transfer is creating new behavioural knowledge from a shared conceptual knowledge. When the source behavioural knowledge, transfered by analogical mechanisms, is not sufficient to solve the target problem, some complementary target behavioural knowledge has to be created. This complementary knowledge results from contextualization of conceptual knowledge. Most work in machine learning simulates this kind of expertise development. A problem solution, obtained by activating conceptual knowledge, forms new behavioural knowledge. In Van De Velde's (1988) approach, this new rule is first over-generalised and then progressively refined (this stage being obviously the most sensitive). Explanation-based learning (EBL) applies these principles to concept acquisition: conceptual knowledge is some declarative and functional description of a concept, learning involves mapping the functional representation to the perceptive description of one instance of this concept (Dejong and Mooney,1986; Mitchell et al.,1986). The functional description is used to determine which perceptive features are relevant for classifying this object as an example of the concept. The resulting behavioural knowledge is a perceptive description of a sub-class of the concept. (The extent to which this subclass represents the concept is again a matter of generalisation/ discrimination techniques, and depends on the careful choice of the worked out example). Dillenbourg et al. Training Transfer Draft for APPLICA 90 p.5 4. Transfer and metacognition. The EBL approach found empirical support in the work of Chi and her colleagues (1989) on self-explanation. These authors gave their subjects the conceptual knowledge in a declarative form. They observed that self-explanations enable their subjects to construct situation-specific rules. As in EBL, these rules explain a particular example in the light of some conceptual knowledge. Interestingly, some rules "seem to be generated with the constraint of the general principle" (Chi et al.,1989, p.179). Subjects complete the conceptual level by inference rules that relate it to the behavioural knowledge. In his work on second generation expert systems, Steels (1989) emphasized that the "deep knowledge" includes some inference component. This ability to construct inference rules determines transfer, since they should enable use of conceptual knowledge for producing target behavioural knowledge. It is hence worth to think about how an ITSs may help to develop it. Not surprisingly, Chi et al observe that this ability is related to the subject's capacity to monitor her own understanding. Brown (1987) confirms this relationship between metacognition and transfer. She associates the concepts of multiple access (the key to transfer) with the concept of reflective access (the key to metacognition). She reports on-going work on the transfer between isomorphic problems by threeand four-year-old children. She found they performed a better transfer when, after each problem, they were required to describe their solution to Kermit the frog so that he could reproduce it. Multiplicity and reflection hence become central issues in the design of ITS. We develop this idea in the next section. 5. Hints to ITS designers The previous section describes two features that the designer should consider to promote transfer : multiplicity and reflection. Multiplicity is intuitively the cornerstone of transfer. Trainees will apply more easily to new contexts some knowledge they have already applied to multiple situations. Given the cost of implementing an environment, repeating exercises within the same contextual pattern has generally been promoted. There is a growing interest in the AI and Education community around the idea of multiple representation of knowledge, representation inside the computer (e.g. in the system QUEST, White and Frederiksen, 1988) or representation on the screen (MacArthur and al., 1988; Leblanc,1988). Greater emphasis should now be placed on the need of varying considerably the problem context (Sack, 1989). In the studies quoted in the previous section, reflection is associated with verbalization. Constraining the trainee to explain her solution or to justify her choices also became more frequent in recent ITSs (e.g. in the system EPIC, Twidale, 1989). This is easy to implement (some basic text processing facilities are required), as far as the system does not attempt to understand the trainee's productions. This is somewhat constraining for the trainee but it can be justified by some artefacts. One of these artefacts consists in adding another trainee, learning with the first one. This second learner may be a real trainee or a computerized colearner played by the system (Dillenbourg and Self, to appear). In this case, verbalizing knowledge and metaknowledge is an essential aspect of the communication which supports the collaboration between learners. However, Brown (1987) insists that verbalization does not always have a positive effect. There is sometimes a negative effect, when the verbalization activity interferes with the described on-going cognitive process. Fortunately, computers have the potential to help the trainee reflect on some graphical representation of her process. This graphical representation reifies abstract features of the user's Dillenbourg et al. Training Transfer Draft for APPLICA 90 p.6 reasoning (Collins and Brown, 1988). The classical example is the Geometry Tutor (Anderson,1985) which represents as a tree the trainee's problem solving path when constructing the proof of some logic theorem. At the same time, ITSs can ease the self-monitoring process by reducing the cognitive load placed on the trainee. Brown (1987) reports that verbalization may have some negative effect on the task performance when it is performed concurrently with the task itself (instead of after task completion). This obvious explanation is the difficulty of performing two tasks simultaneously. An ITS may assume some routinized aspects of the task in order to free a part of the learner's resources. These resources are then expected to be available for better selfmonitoring. For example, Algebraland (Foss,1987) performs some algebraic manipulations, and hence gives the learner the opportunity to pay more attention to strategic aspects of the solution.